This invention relates to the field of computer systems. More particularly, a system and methods are provided for testing software in an adaptive manner, to provide maximum test coverage in a minimum period of time.
In software engineering, a great deal of effort is necessarily dedicated to testing software, and the testing methods and strategy have a direct impact on the quality of the end product. Traditional blackbox and whitebox methods for testing software generally provide quality and coverage information only during the test development period. After the test development cycle has been completed, test execution alone generally fails to instill full confidence that the software under test has been thoroughly and accurately examined. This is because most software, after its initial release, will still require dynamic code changes for optimization, additional functionality, bug fixes, etc.
Currently, most common testing techniques still include manual testing and/or the use of static test scripts. Although a software tester may become familiar with particular test scripts or cases, they are often not updated as the software being tested evolves. Thus, the test cases that are applied, and the software components they are applied against, often do not match well over time. As a result, it is difficult to achieve clear, effective and systematic test planning, execution and control.
In addition, many test cases or scripts grow in uncontrolled (or at least undocumented) fashion. Over time, they may become complex, unwieldy and difficult to follow as they are manually modified to attempt to match evolving software. And, modifications often lack objectivity, due to time constraints or lack of knowledge of the software. Thus, any given test case may become so obsolete that it no longer fully tests the functionality of a program, or does not test the functionality that needs to be tested or that is expected to be tested.
Furthermore, most test processes will perform a full test cycle for every software change, even if the test cycle tests far more than is needed. For example, even if only one component or one function of a modular set of software needs to be tested, test cases for testing that component and test cases for testing many other components or functions will all be executed. This wastes time and may prevent the software from being released on schedule.
Also, test cases or scripts may become more and more redundant or obsolete over time, as the bugs they are intended to catch are caught and fixed. The test cases or scripts may continue to be run in their entirety, to test for bugs that will no longer be found, because there is no automated mechanism to allow more accurate matching of tests and software components or functions.
In general, it is desirable to employ tests that will have the highest probability of finding the most errors, with a minimal amount of time and effort. However, after a period of time, existing test cases or scripts may no longer correlate well with ever-changing target software components. Thus, it is often not known which portion of the software will be tested by which test cases. Commercial testing and debugging tools cannot identify, for a given portion of software, which test cases will test that portion. They also do not provide any rating of how well a test case covers a software component; they generally treat all test cases as being equal. During test execution, it is important to know which test cases are affected by which software changes.
Also, it is often physically impossible to test every permutation or aspect of a program. There may be only a limited amount of time available for testing. At some point, a decision may need to be made as to how much testing to perform, or whether sufficient testing has been performed. When it is not fully known how much of the software that needs to be tested has been tested (e.g., because test cases are not well understood), it can be very difficult or even impossible to make a correct go/no-go decision.
Thus, there is a need for a method of testing software that provides justifiable confidence that software components or functions that need to be tested have been tested, and that test cases applied during testing actually tested the components or functionality that need to be tested.